The present disclosure relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program, by which it is possible to generate a three-dimensional image (a 3D image) which can be stereoscopically viewed.
An image corresponding to stereoscopic vision visible as a stereoscopic image having a depth includes a combination of two images, that is, a left eye image and a right eye image, which are images from different viewpoints. In order to obtain these images from two viewpoints, that is, a binocular parallax image, for example, two imaging apparatuses are separated from each other from side to side to perform imaging.
A pair of captured stereoscopic images include pair images, that is, a left eye image captured by a left imaging apparatus and observed by the left eye and a right eye image captured by a right imaging apparatus and observed by the right eye.
The stereoscopic image pair including the pair of the left eye image and the right eye image is displayed on a display apparatus capable of providing the left eye image and the right eye image to the left eye and the right eye of an observer by separating the left eye image from the right eye image, so that it is possible for the observer to view the image as a stereoscopic image.
Meanwhile, in the related art, there have been proposed various configurations for generating a binocular parallax image including a left eye image and a right eye image corresponding to stereoscopic vision using a normal two-dimensional image photographed from one viewpoint.
For example, Japanese Unexamined Patent Application Publication No. 8-30806 has disclosed an apparatus which shifts a left eye image and a right eye image in a horizontal direction by a predetermined amount with respect to a still image or an image with small motion, thereby allowing the image to be recognized as standing out.
Furthermore, Japanese Unexamined Patent Application Publication No. 2005-151534 has disclosed a method of calculating feature amounts of an upper portion and a lower portion of an image and adjusting a combination rate of a plurality of scene structures indicating depth information prepared in advance, thereby expressing an image by a combination of simple structures.
When a stereoscopic image is captured using a binocular imaging system, a base line length and a convergence angle are important factors for determining a binocular parallax value. The base line length denotes a distance between two imaging apparatuses and the convergence angle denotes directions of the two imaging apparatuses.
FIG. 1 illustrates stereoscopic imaging systems having different convergence angles based on the same base line length.
In three binocular imaging systems (a), (b), and (c) illustrated in FIG. 1, the base line lengths are all L and the convergence angles are parallel view, 15°, and 30°.
As the base line length is increased, a binocular parallax value of the right and left images is increased. The convergence angle corresponds to a point at which the binocular parallax value of the right and left images is 0 with respect to an object positioned at an intersection point of convergence, and a binocular parallax of both eyes is increased the farther an object is from the intersection point of convergence.
Therefore, when an object is far from the imaging system (a camera) as illustrated in FIG. 1(a), imaging is properly performed in the state in which the base line length is long and the convergence angle is small (approximate to parallel view).
Meanwhile, when an object is near the imaging system as illustrated in FIG. 1(c), it is commonly considered that imaging is properly performed in the state in which the base line length is short and the convergence angle is large.
However, adjusting the base line length and the convergence angle for each object to be captured is problematic because much time and effort are necessary. Although this is possible for professional imaging in which time may be spent for the adjustment, in general imaging use, fixed base line length and convergence angle are generally used. In addition, in order to make the binocular imaging apparatuses suitable for imaging an object in a wider range, there are many binocular imaging apparatuses configured to have conditions in which the base line length has a predetermined value or more and the convergence angle is as small as possible. These imaging apparatuses are not suitable for imaging an object at a near distance and various problems occur at the time of the imaging as described above.
FIG. 2 illustrates an example of stereoscopic imaging performed by a binocular imaging system A (10) and a binocular imaging system B (20) which have different distances from an object.
The binocular imaging system A (10) is far the object and the binocular imaging system B (20) is near from the object. In both of the imaging systems, a base line length and a convergence angle are equal to each other.
FIG. 3 illustrates images as imaging results by the binocular imaging system A (10) and the binocular imaging system B (20) illustrated in FIG. 2.
FIG. 3(a) illustrates images captured by the binocular imaging system A (10) and FIG. 3(b) illustrates images captured by the binocular imaging system B (20).
FIG. 3(b), which illustrates the images captured by the binocular imaging system B (20) near the object, shows a large parallax value between the right and left images, as compared with FIG. 3(a) which illustrates the images captured by the binocular imaging system A (10) far from an object. This is apparent if areas at which the objects are positioned in the images are compared with each other from side to side.
The images illustrated in FIG. 3(b) are in a state in which a retinal image difference is very large, and cause discomfort and fatigue when the images are observed.
Moreover, in the images illustrated in FIG. 3(b), areas not reflected on one of the left eye image and the right eye image are reflected on the other one, that is, many occlusion areas are generated, as compared with the two images illustrated in FIG. 3(a).
For example, in the image pair illustrated in FIG. 3(b), the lights of a car or the windows of a building shown in the left eye image are not shown in the right image. Furthermore, the trunk of a tree shown in the right eye image is not shown in the left image, and many such occlusion areas are generated.
As illustrated in FIG. 3(b), when an image in which the difference between the right and left retinal images has a predetermined value or more, or an image in which occlusion areas are generated is observed by an observer through stereoscopic vision, the observer may feel discomfort or fatigue of eyes in many cases. On the other hand, when such a phenomenon is minimal as illustrated in FIG. 3(a), comfortable stereoscopic vision can be obtained.
The processes described with reference to FIG. 1 to FIG. 3 correspond to processing examples when photographed images from two different viewpoints are used as stereoscopic vision images.
Meanwhile, as simply described above, there is a configuration in which two images having pseudo-parallax are generated by image processing based on one image, and are used as stereoscopic vision images. That is, it denotes a process in which a 2D/3D conversion process is performed on one 2D image photographed by a general camera to generate a left eye image and a right eye image.
Since the 2D/3D conversion processes generate right and left parallax images from information on one image, there is a general problem that a retinal image difference is small and a stereoscopic effect recognized by an observer is reduced. However, there is a low probability of discomfort or fatigue of eyes due to a large retinal image difference or an occlusion area.
FIG. 4 is a diagram illustrating a process for performing a 2D/3D conversion process on one left eye image photographed by the binocular imaging system B (20) illustrated in FIG. 2 to generate a left eye image and a right eye image as stereoscopic images, and the processing result.
FIG. 4(b) at an upper portion of FIG. 4 is the same as FIG. 3(b) and illustrates two images captured by the binocular imaging system B (20) illustrated in FIG. 2. Meanwhile, FIG. 4(c) at a lower portion of FIG. 4 illustrates the left eye image and the right eye image generated by performing the 2D/3D conversion process on the left eye image which is one image of FIG. 4(b).
The left eye image and the right eye image of FIG. 4(c) generated through the 2D/3D conversion process have no large retinal image difference or occlusion area, as compared with the left eye image and the right eye image of FIG. 4(b) photographed by the binocular imaging system B (20).
An observer observing the image illustrated in FIG. 4(c) can feel comfortable stereoscopic vision without feeling fatigue or discomfort.
As described above, when an imaging apparatus is near an object, if an image photographed by the imaging apparatus is used as a stereoscopic image without any image conversion process, an observer may feel discomfort or fatigue. Meanwhile, when an image generated through the 2D/3D conversion process (stereoscopic conversion) for converting a two-dimensional image to a binocular parallax image is used, it is possible to adjust parallax and to generate an image not causing the above-mentioned discomfort or fatigue.
However, an image generated from one image through the 2D/3D conversion process is an image having pseudo-parallax. For example, when an imaging apparatus is positioned at a predetermined distance or more from an object, display is performed using a stereoscopic image photographed using a binocular imaging system, thereby obtaining an image correctly reflecting an actual distance and achieving comfortable stereoscopic vision.
As described above, depending on the situation, either of display, without any image conversion process, using a left eye image and a right eye image photographed by a binocular imaging system and display using a 2D/3D conversion image generated from one image may be preferable.